feat: Add WASM-based python processor #621

gregfurman · 2025-12-18T21:19:55Z

Motivation

Includes a new python processor that runs within a sandboxed WASM environment.

Changes

Adds a new python processor that runs within a sandboxed WASM environment.
Adds a WasmModulePool in internal/impl/wasm/wasmpool/pool.go that uses a Golang sync.Pool to execute multiple concurrent guests and have their lifecycles managed by the GC. This allows us to spin up more instances as demand increases and have the Go GC handle the guest lifecycle.
Adds an entrypoint.py that allows python execution script to compile and execute -- stdout and stdin is used to pass data between the guest and host.
If the x_wasm tag is used, the WASM executable and corresponding entrypoint.py are embedded at compile time. Using the processor without this tag panics on init.
Adds a script at scripts/install.sh that downloads the python-3.12.0.wasm and checks it against the SHA in runtime/python-3.12.0.wasm.sha256sum.

Installation/Usage

Run the following commands to create a config, install the python runtime (and check it against the SHA), and execute a pipeline.

cat > python.yaml <<'EOF'
pipeline:
  processors:
    - python:
        script: |
          root = sum(this)
EOF

bash internal/impl/python/scripts/install.sh

echo "[11, 6, 20, 5]" | go run -tags "x_wasm" cmd/bento/main.go -c python.yaml --set logger.level=warn

The output of the pipeline should be 42.

TODO

Extend the CI (or Makefile) to run the downloader script and build with the x_wasm tag -- allowing the .wasm to be embedded at compile time and used in the processor.

aronchick · 2025-12-20T16:39:06Z

this is going to be really cool! lmk if you need help

gregfurman · 2025-12-21T11:51:32Z

Thanks @aronchick! If you have time (and are feeling extra generous), I'd appreciate if you could run through the Installation/Usage steps locally and try out different configurations. I'm also looking for any feedback on usability since we're planning to expand our WASM offering, so any insights would be super helpful!

I haven't dug too deep into the limitations of the sandboxed runtime and how it responds to accessing non-permissive resources (e.g., filesystem or OS).

In addition, we'd probably need to soak test the processor before making this GA in Bento since I'm unsure of performance/behavior over longer periods and under differing load profiles.

gregfurman

@jem-davies I've added a quick self-review. Let me know if you think we should be adding more documentation with what's been mentioned here.

gregfurman · 2025-12-21T11:56:18Z

internal/impl/python/runtime/entrypoint.py

+            namespace = {"this": input_data, "root": None, "__builtins__": __builtins__, **injected_libs}
+            exec(compiled_code, namespace)


this namespace and exec combintation allows us to dynamically run our compiled execution script, constraining what is accessible during runtime and ensuring no global state is ever mutated.

Also, see how we pass this and root -- leveraging the same pattern we use with bloblang scripts/mappings.

gregfurman · 2025-12-21T11:57:02Z

internal/impl/python/runtime/entrypoint.py

+        name = lib_name.strip()
+        if name:
+            try:
+                injected[name] = importlib.import_module(name)


Some python magic that allows us to dynamically extract which libraries we're making permissable to the sandboxed execution.

gregfurman · 2025-12-21T11:58:47Z

internal/impl/python/scripts/install.sh

+
+if command -v sha256sum >/dev/null; then
+    (cd "$WASM_RUNTIME_DIR/runtime" && sha256sum --check --status python-3.12.0.wasm.sha256sum)
+elif command -v shasum >/dev/null; then


fallback to shasum since sha256sum is not available on darwin

gregfurman · 2025-12-21T12:03:33Z

internal/impl/python/scripts/install.sh

+wget -q "$WASM_BINARY_URL" -O "$WASM_PATH"
+
+if command -v sha256sum >/dev/null; then
+    (cd "$WASM_RUNTIME_DIR/runtime" && sha256sum --check --status python-3.12.0.wasm.sha256sum)


Since we're downloading and embedding a WASM executable into bento, we need to verify the integrity of the executable we're intending to use. So we download and commit the SHA256 checksum, then verify the downloaded file's checksum.

gregfurman · 2025-12-21T12:05:12Z

internal/impl/wasm/wasmpool/pool.go

+	instancePool sync.Pool
+}
+
+func NewWasmModulePool[T api.Module](ctx context.Context, ctor constructor[T]) (*WasmModulePool[T], error) {


Unfortunately WASM modules (at least with wazero) are not safe for concurrent executions. Instead, we create a module pool (backed by a sync.Pool) and allow the GC to dynamically spin up instances depending on load.

gregfurman · 2025-12-21T12:08:03Z

internal/impl/wasm/wasmpool/pool.go

+}
+
+func (p *WasmModulePool[T]) Put(instance T) {
+	_ = runtime.AddCleanup(&instance, func(inst T) {


If the GC cleans up the instance, which is possible given sync.Pool, we need to ensure the resources are actually freed up. Hence, this gross runtime.AddCleanup hook that triggers when the module is GC'd

gregfurman · 2025-12-21T12:11:18Z

internal/impl/wasm/wasmpool/pool.go

+	p.instancePool.Put(instance)
+}
+
+func (p *WasmModulePool[T]) Close(ctx context.Context) error {


While it's probably easier to just close the entire runtime to ensure all associated modules (and resources) are cleaned up, this Close will retrieve all objects from the pool and close them.

While thinking of this as a Reset or Cleanup conceptually more correct, Close feels more idiomatic 🤷

gregfurman · 2025-12-21T12:15:04Z

internal/impl/python/instance.go

+	}()
+
+	handshakeErr := make(chan error, 1)
+	go func() {


We do a quick-and-dirty handshake to ensure the python env is up-and-running. The python module should write a "READY" signal to the output socket which we need to check is received before proceeding.

gregfurman · 2025-12-21T12:16:24Z

internal/impl/python/instance.go

+		return nil, pi.stderrBuf.Bytes(), err
+	}
+
+	header := make([]byte, 5)


First byte signals that the response was a success or exception. Last four bytes tell us how long the response is.

aronchick · 2025-12-22T00:23:15Z

Testing Instructions

I've tested this locally and it works great! Here's how others can try it:

Quick Start

# 1. Checkout the PR
gh pr checkout 621

# 2. Download the Python WASM runtime (~50MB) and test configs
bash internal/impl/python/scripts/install.sh
gh gist clone https://gist.github.com/aronchick/5d9391b03f9266b994c7a36043d743a4 python-examples

# 3. Run an example (first build takes ~30-60s)
go run -tags "x_wasm" ./cmd/bento/main.go -c python-examples/hello.yaml

Expected output: "Hello, World!"

Example Configs

Full test configs available in the gist: <GIST_URL>

File	Description
`hello.yaml`	Simple greeting
`test1.yaml`	JSON transformation
`test2.yaml`	Array processing (sum, count, double)
`test3.yaml`	Message filtering
`test4.yaml`	Stdlib imports (math, json, re)
`test5.yaml`	E-commerce order transformation
`test6.yaml`	Error handling demo

Quick Snippets

Array processing:

- python:
    script: |
      root = {"sum": sum(this), "doubled": [x * 2 for x in this]}

With imports:

- python:
    imports: [math, json, re]
    script: |
      root = {"ceil": math.ceil(this["value"])}

Filtering (set root=None to drop):

- python:
    script: |
      root = this if this.get("status") == "active" else None

Run Unit Tests

go test -tags "x_wasm" -v ./internal/impl/python/...

All 18 tests pass (~18 seconds).

Key Points

Build tag required: Always use -tags "x_wasm"
Cold start: ~1 second per Python instance
Familiar syntax: Uses this and root like Bloblang
Sandboxed: Runs in WASM, no filesystem/network access

jem-davies · 2026-01-17T16:18:25Z

Question,

I see you have a TODO:

Extend the CI (or Makefile) to run the downloader script and build with the x_wasm tag -- allowing the .wasm to be embedded at compile time and used in the processor.

Is the plan then that for the binaries we publish to the release page be embedded with the .wasm?

Just trying to put myself in place of a user at the moment and am seeing that I need to follow instructions on the PR to get the processor to work - but they aren't on the doc page for this new processor?

gregfurman · 2026-01-17T16:36:41Z

Question,

I see you have a TODO:

Extend the CI (or Makefile) to run the downloader script and build with the x_wasm tag -- allowing the .wasm to be embedded at compile time and used in the processor.

Is the plan then that for the binaries we publish to the release page be embedded with the .wasm?

Just trying to put myself in place of a user at the moment and am seeing that I need to follow instructions on the PR to get the processor to work - but they aren't on the doc page for this new processor?

Thanks for giving this a look!

Yeah so the plan is to embed this binary inside the executable. We can also zip the bin with brotli (which makes it much smaller) and just unzip at runtime if the processor is used.

Wouldn't expect a user to run any of this.

Ideally would want to drop the x_wasm tag and allow this in the standard built.

Thoughts?

jem-davies · 2026-01-18T15:40:15Z

Ideally would want to drop the x_wasm tag and allow this in the standard built.

Yeah, that is what I was thinking 👍

jem-davies

Looks really good - I think I would just ship it as experimental not stable - for a few months.

Also would be interesting to see how we will update the CI/CD to deal with the inclusion of the things we need to embed - and how that will look on different OS / Cpu Arch? (Which I assume could be a problem...)

I think the WASM approach is great - we just need to finish the TODO re. CI

gregfurman · 2026-01-20T19:44:22Z

@jem-davies OK have updated this to use make generate with an associated downloader script in runtime/install.go.

This also allows the downloading to be compatible with windows builds since the .sh has been replaced with a go script that both retrieves and validates the binary.

We just need to add make generate to the goreleaser process to ensure the .wasm is present 👍

Question though: are we OK with the binary being downloaded each time we need to run a CI job? I see linting is (rightfully) failing now since the embed directive cannot find the non-existent file .wasm and entrypoint.py files

jem-davies · 2026-02-05T16:44:35Z

internal/impl/python/processor.go

+
+func pythonProcessorSpec() *service.ConfigSpec {
+	return service.NewConfigSpec().
+		Stable().


Thoughts about downgrading to 'experimental' and then getting this merged?

gregfurman · 2026-02-12T09:26:11Z

@jem-davies OK will merge!

FYI Pydantic recently released https://github.com/pydantic/monty which is basically a python subset written in Rust. I ran some tests locally and managed to compile it to a WASM bundle and it's only like 4mb uncompressed. Will look into replacing this VMWare binary at some point with it instead.

Tbh I'm not the biggest fan of embedding a zip into the go binary but 🤷

gregfurman self-assigned this Dec 18, 2025

gregfurman force-pushed the add/processor/python branch from 5c82ab2 to 38217e5 Compare December 18, 2025 21:23

gregfurman force-pushed the add/processor/python branch 2 times, most recently from b0dd215 to f212bc8 Compare December 20, 2025 17:59

gregfurman marked this pull request as ready for review December 20, 2025 21:11

gregfurman requested a review from jem-davies as a code owner December 20, 2025 21:11

gregfurman commented Dec 21, 2025

View reviewed changes

jem-davies reviewed Jan 19, 2026

View reviewed changes

gregfurman force-pushed the add/processor/python branch from 90e96d9 to fd6374e Compare January 21, 2026 15:31

gregfurman added 4 commits January 21, 2026 16:06

feat: Add WASM-based python processor

0010b4a

update: Remove custom build tag and use go generate for downloading

7adb470

add to CI and rework to allow for .wasm not existing

3d9c56a

use in-memory compilation cache

d2ef532

gregfurman force-pushed the add/processor/python branch from 4257b03 to d2ef532 Compare January 21, 2026 16:06

gregfurman added 2 commits January 21, 2026 16:09

fix docs

f91ece3

add: go generate to dockerfile steps

6daecfa

gregfurman requested a review from jem-davies January 21, 2026 16:51

revert approach and embed .zip

e391b77

jem-davies approved these changes Feb 5, 2026

View reviewed changes

chore: update docs

cc8a1a9

gregfurman merged commit 70c1413 into warpstreamlabs:main Feb 12, 2026
3 checks passed

		namespace = {"this": input_data, "root": None, "__builtins__": __builtins__, **injected_libs}
		exec(compiled_code, namespace)

feat: Add WASM-based python processor #621

feat: Add WASM-based python processor #621

Uh oh!

Conversation

gregfurman commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Installation/Usage

TODO

Uh oh!

aronchick commented Dec 20, 2025

Uh oh!

gregfurman commented Dec 21, 2025

Uh oh!

gregfurman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aronchick commented Dec 22, 2025

Testing Instructions

Quick Start

Example Configs

Quick Snippets

Run Unit Tests

Key Points

Uh oh!

jem-davies commented Jan 17, 2026

Uh oh!

gregfurman commented Jan 17, 2026

Uh oh!

jem-davies commented Jan 18, 2026

Uh oh!

jem-davies left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregfurman commented Jan 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregfurman commented Feb 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gregfurman commented Dec 18, 2025 •

edited

Loading

jem-davies left a comment •

edited

Loading